Incorporating Season and Solar Specificity Into Renderings Made by a NeRF Architecture Using Satellite Images.

As a result of Shadow NeRF and Sat-NeRF, it is possible to take the solar angle into account in a NeRF-based framework for rendering a scene from a novel viewpoint using satellite images for training. Our work extends those contributions and shows how one can make the renderings season-specific. Our main challenge was creating a Neural Radiance Field (NeRF) that could render seasonal features independently of viewing angle and solar angle while still being able to render shadows. We teach our network to render seasonal features by introducing one more input variable - time of the year. However, the small training datasets typical of satellite imagery can introduce ambiguities in cases where shadows are present in the same location for every image of a particular season. We add additional terms to the loss function to discourage the network from using seasonal features for accounting for shadows. We show the performance of our network on eight Areas of Interest containing images captured by the Maxar WorldView-3 satellite. This evaluation includes tests measuring the ability of our framework to accurately render novel views, generate height maps, predict shadows, and specify seasonal features independently from shadows. Our ablation studies justify the choices made for network design parameters.

A Laplacian Pyramid Based Generative H&E Stain Augmentation Network.

Hematoxylin and Eosin (H&E) staining is a widely used sample preparation procedure for enhancing the saturation of tissue sections and the contrast between nuclei and cytoplasm in histology images for medical diagnostics. However, various factors, such as the differences in the reagents used, result in high variability in the colors of the stains actually recorded. This variability poses a challenge in achieving generalization for machine-learning based computer-aided diagnostic tools. To desensitize the learned models to stain variations, we propose the Generative Stain Augmentation Network (G-SAN) - a GAN-based framework that augments a collection of cell images with simulated yet realistic stain variations. At its core, G-SAN uses a novel and highly computationally efficient Laplacian Pyramid (LP) based generator architecture, that is capable of disentangling stain from cell morphology. Through the task of patch classification and nucleus segmentation, we show that using G-SAN-augmented training data provides on average 15.7% improvement in F1 score and 7.3% improvement in panoptic quality, respectively. Our code is available at https://github.com/lifangda01/GSAN-Demo.

DEBISim: A simulation pipeline for dual energy CT-based baggage inspection systems1.

BACKGROUND: Materials characterization made possible by dual energy CT (DECT) scanners is expected to considerably improve automatic detection of hazardous objects in checked and carry-on luggage at our airports. Training a computer to identify the hazardous items from DECT scans however implies training on a baggage dataset that can represent all the possible ways a threat item can packed inside a bag. Practically, however, generating such data is made challenging by the logistics (and the permissions) related to the handling of the hazardous materials. OBJECTIVE: The objective of this study is to present a software simulation pipeline that eliminates the need for a human to handle dangerous materials and that allows for virtually unlimited variability in the placement of such materials in a bag alongside benign materials. METHODS: In this paper, we present our DEBISim software pipeline that carries out an end-to-end simulation of a DECT scanner for virtual bags. The key highlights of DEBISim are: (i) A 3D user-interactive graphics editor for constructing a virtual 3D bag with manual placement of different types of objects in it; (ii) An automated virtual bag generation algorithm for creating randomized baggage datasets; (iii) An ability to spawn deformable sheets and liquid-filled containers in a virtual bag to represent plasticized and liquid explosives; and (iv) A GPU-based X-ray forward modelling block for spiral cone-beam scanners used in checked baggage screening. RESULTS: We have tested our simulator using two standard CT phantoms: the American College of Radiology (ACR) phantom and the NIST security screening phantom as well as on a set of reference materials representing commonly encountered items in checked baggage. For these phantoms, we have assessed the quality of the simulator by comparing the simulated data reconstructions with real CT scans of the same phantoms. The comparison shows that the material-specific properties as well as the CT artifacts in the scans generated by DEBISim are close to those produced by an actual scanner. CONCLUSION: DEBISim is an end-to-end simulation framework for rapidly generating X-ray baggage data for dual energy cone-beam scanners.

Cluster-based distributed face tracking in camera networks.

In this paper, we present a distributed multicamera face tracking system suitable for large wired camera networks. Unlike previous multicamera face tracking systems, our system does not require a central server to coordinate the entire tracking effort. Instead, an efficient camera clustering protocol is used to dynamically form groups of cameras for in-network tracking of individual faces. The clustering protocol includes cluster propagation mechanisms that allow the computational load of face tracking to be transferred to different cameras as the target objects move. Furthermore, the dynamic election of cluster leaders provides robustness against system failures. Our experimental results show that our cluster-based distributed face tracker is capable of accurately tracking multiple faces in real-time. The overall performance of the distributed system is comparable to that of a centralized face tracker, while presenting the advantages of scalability and robustness.

3D modeling of optically challenging objects.

We present a system for constructing 3D models of real-world objects with optically challenging surfaces. The system utilizes a new range imaging concept called multi-peak range imaging, which stores multiple candidates of range measurements for each point on the object surface. The multiple measurements include the erroneous range data caused by various surface properties that are not ideal for structured-light range sensing. False measurements generated by spurious reflections are eliminated by applying a series of constraint tests. The constraint tests based on local surface and local sensor visibility are applied first to individual range images. The constraint tests based on global consistency of coordinates and visibility are then applied to all range images acquired from different viewpoints. We show the effectiveness of our method by constructing 3D models of five different optically challenging objects. To evaluate the performance of the constraint tests and to examine the effects of the parameters used in the constraint tests, we acquired the ground truth data by painting those objects to suppress the surface-related properties that cause difficulties in range sensing. Experimental results indicate that our method significantly improves upon the traditional methods for constructing reliable 3D models of optically challenging objects.

Error analysis of robust optical flow estimation by least median of squares methods for the varying illumination model.

The apparent pixel motion in an image sequence, called optical flow, is a useful primitive for automatic scene analysis and various other applications of computer vision. In general, however, the optical flow estimation suffers from two significant problems: the problem of illumination that varies with time and the problem of motion discontinuities induced by objects moving with respect to either other objects or with respect to the background. Various integrated approaches for solving these two problems simultaneously have been proposed. Of these, those that are based on the LMedS (Least Median of Squares) appear to be the most robust. The goal of this paper is to carry out an error analysis of two different LMedS-based approaches, one based on the standard LMedS regression and the other using a modification thereof as proposed by us recently. While it is to be expected that the estimation accuracy of any approach would decrease with increasing levels of noise, for LMedS-like methods, it is not always clear as to how much of that decrease in performance can be attributed to the fact that only a small number of randomly selected samples is used for forming temporary solutions. To answer this question, our study here includes a baseline implementation in which all of the image data is used for forming motion estimates. We then compare the estimation errors of the two LMedS-based methods with the baseline implementation. Our error analysis demonstrates that, for the case of Gaussian noise, our modified LMedS approach yields better estimates at moderate levels of noise, but is outperformed by the standard LMedS method as the level of noise increases. For the case of salt-and-pepper noise, the modified LMedS method consistently performs better than the standard LMedS method.

A deformable model for automatic CT liver extraction.

RATIONALE AND OBJECTIVES: This study was performed to design an automatic liver region extraction system to facilitate clinical liver size estimation and further serve as a prestage for liver reconstruction and volume estimation. MATERIALS AND METHODS: We present a modification of the well-known snakes algorithm for extracting liver regions in noisy CT images. Our modification addresses the issues of selection of the control points on an estimate of the contour and the determination of the weighting coefficients. The weighting coefficients are determined dynamically on the basis of the distance between the control points and the local curvature of the contour. RESULTS: The proposed method was used in extracting liver regions from 98 cross-sectional abdominal images. The overall performance was estimated by comparisons with original liver regions. CONCLUSION: The deformable model method enables an efficient and effective automatic liver region extraction in noisy environments. This approach eliminates human-in-the loop, which is the common practice for the majority of current methods.

A subsumptive, hierarchical, and distributed vision-based architecture for smart robotics.

We present a distributed vision-based architecture for smart robotics that is composed of multiple control loops, each with a specialized level of competence. Our architecture is subsumptive and hierarchical, in the sense that each control loop can add to the competence level of the loops below, and in the sense that the loops can present a coarse-to-fine gradation with respect to vision sensing. At the coarsest level, the processing of sensory information enables a robot to become aware of the approximate location of an object in its field of view. On the other hand, at the finest end, the processing of stereo information enables a robot to determine more precisely the position and orientation of an object in the coordinate frame of the robot. The processing in each module of the control loops is completely independent and it can be performed at its own rate. A control Arbitrator ranks the results of each loop according to certain confidence indices, which are derived solely from the sensory information. This architecture has clear advantages regarding overall performance of the system, which is not affected by the "slowest link," and regarding fault tolerance, since faults in one module does not affect the other modules. At this time we are able to demonstrate the utility of the architecture for stereoscopic visual servoing. The architecture has also been applied to mobile robot navigation and can easily be extended to tasks such as "assembly-on-the-fly."

An interactive framework for acquiring vision models of 3-D objects from 2-D images.

This paper presents a human-computer interaction (HCI) framework for building vision models of three-dimensional (3-D) objects from their two-dimensional (2-D) images. Our framework is based on two guiding principles of HCI: 1) provide the human with as much visual assistance as possible to help the human make a correct input; and 2) verify each input provided by the human for its consistency with the inputs previously provided. For example, when stereo correspondence information is elicited from a human, his/her job is facilitated by superimposing epipolar lines on the images. Although that reduces the possibility of error in the human marked correspondences, such errors are not entirely eliminated because there can be multiple candidate points close together for complex objects. For another example, when pose-to-pose correspondence is sought from a human, his/her job is made easier by allowing the human to rotate the partial model constructed in the previous pose in relation to the partial model for the current pose. While this facility reduces the incidence of human-supplied pose-to-pose correspondence errors, such errors cannot be eliminated entirely because of confusion created when multiple candidate features exist close together. Each input provided by the human is therefore checked against the previous inputs by invoking situation-specific constraints. Different types of constraints (and different human-computer interaction protocols) are needed for the extraction of polygonal features and for the extraction of curved features. We will show results on both polygonal objects and object containing curved features.

Automated storage and retrieval of thin-section CT images to assist diagnosis: system description and preliminary assessment.

A software system and database for computer-aided diagnosis with thin-section computed tomographic (CT) images of the chest was designed and implemented. When presented with an unknown query image, the system uses pattern recognition to retrieve visually similar images with known diagnoses from the database. A preliminary validation trial was conducted with 11 volunteers who were asked to select the best diagnosis for a series of test images, with and without software assistance. The percentage of correct answers increased from 29% to 62% with computer assistance. This finding suggests that this system may be useful for computer-assisted diagnosis.